AI-Driven Performance Modeling for AI Inference Workloads

Deep Learning (DL) is moving towards deploying workloads not only in cloud datacenters, but also to the local devices. Although these are mostly limited inference tasks, it still widens range of possible target architectures significantly. Additionally, new targets usually come with drastically reduced computation performance and memory sizes compared traditionally used architectures—and put key optimization focus on efficiency as they often depend batteries. To help developers quickly estimate a neural network during its design phase, models could be used. However, expensive implement require in-depth knowledge about hardware architecture algorithms. AI-based solutions exist, either large datasets that difficult collect low-performance and/or small number platforms metrics. Our solution exploits block-based structure networks, well high similarity typically layer configurations across enabling training accurate significantly smaller datasets. In addition, our specific or metric. We showcase feasibility set seven devices from four different architectures, up three metrics per target—including power consumption footprint. tests have shown achieved an error less than 1 ms (2.6%) latency, 0.12 J (4%) energy 11 MiB (1.5%) allocation for whole prediction, while being five orders magnitude faster benchmark.